1. Field of the Invention
The present invention relates generally to the field of testing and specifically to evaluating which software programs and portions of software programs are at the greatest risk of failure if they are not fully tested.
2. Description of the Background Art
Software developers commonly spend a great deal of time and money testing the computer programs they develop. This is especially true for development of so-called “mission critical” programs, such as those used in business accounting, payroll and billing systems.
Oftentimes, managers, developers and testers are unable to verify how well software has been tested in order to predict the likelihood that untested problems may emerge at some later time. Though extensive testing may have been undertaken, significant portions of the program logic may not have been exercised.
Software testing is viewed as an obstacle to production because it is time consuming and the costs of not testing are theoretical and uncertain. Consequently, software testing is often neglected. One way to balance deployment time with higher software quality resulting from thorough testing is to make testing and development as efficient as possible.
As software becomes increasingly more complex, testing the software also becomes more complex as a large number of unique combinations of paths and modules may be tested for each program. Therefore it is difficult to determine how much testing is sufficient to ensure a minimum level of quality in a software program to be shipped. One approach to software testing is to run as many tests as possible in a given time frame. However, this approach is inefficient because all of the tests may not be necessary and the testing is performed indiscriminately. Some other common inefficiencies in software testing include: testing a small portion of the application repetitively using production data, failing to identify and test all of the changed or impacted routines, placing the same priority on testing rarely-used and heavily-used routines, testing impossible paths or dead branches because they are there, and spending time sorting out problems that result from using bad test data.
Previous solutions to the testing efficiency problem have identified one risk factor and calculated one or more metrics that represent the individual risk factor. For example, a first risk metric may show a module's complexity, a second risk metric may indicate the number of changes in a module, and another metric may reveal the percentage of unexecuted statements in the module. If these individual metrics were combined, the combination was accomplished by adding or multiplying the individual risk metrics together with individual weighting factors. The result of this conventional method is to generate a single number that falls within a continuous range of values in which the magnitude of the generated number indicates whether the program being tested is either good or bad. While this method may be sufficient to identify programs that may have problems, the conventional method does not enable a method to let the user determine which factors were present in the program by simply observing the value of the generated number. Additionally, the conventional method does not provide convenient way to group programs having similar problems, an option which is also useful to the software developer.
What is needed is an improved system and method for enabling more efficient testing of computer programs. It would be desirable for this improved system and method to combine individual risk factors to produce a single risk metric based on the combination of factors that also indicates which individual risk factors were present in the program. In particular, it would be desirable for the single risk metric to enable a comparison of the relative riskiness of multiple programs or portions of a program, and to indicate to which individual risk factor the greatest amount of testing efforts should be devoted.